“Whatever you now find weird, ugly, uncomfortable, and nasty about a new medium will surely become its signature.
CD distortion, the jitteriness of digital video, the crap sound of 8-bit, all of these will be cherished and emulated as soon as they can be avoided. It’s the sound of failure.
So much modern art is the sound of things going out of control. Out of a medium, pushing to its limits and breaking apart.”
-- Brian Eno

My goal is to help you understand:
I'm Andrew Look, ML engineer on the Knowledge team at Pinterest.
I work on building classifier algorithms to look at each pin and determine its Taste Graph Interests.
I started painting as a stress-relief hobby from startup life pre-Pinterest.
# asset ideas:
# - hacker companion painting
# - me asleep with book
# - pinterest yearbook
But I have spent a couple years learning about the space, so I'll do my best to explain things as I understand them so far.
If you don't have a laptop, this presentation is designed to still be interesting and useful.
Request: Audience volunteer to slack me an image I can use for demos: @look
If you have a laptop, you may follow along using this interactive notebook hosted on Google Colab: Start Here
All you need to do to follow along are a few steps:
If you want to save your notebook to you own Google Drive account.
In the top menu, choose File -> Save a copy in drive from the top menu when you're ready to save.
This makes notebook run much, much faster.
In the top menu, Click Runtime and then select Change Runtime Type.
Then, click the Hardware Accelerator dropdown, select GPU and then click Save
ipyd.Image(filename='drive/projects/whirlwind/gpu_py2_colab.png', width=400)
When you're done, running the following code cell should confirm that a GPU is available:
import tensorflow as tf
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
import sys
print(sys.version)
assert sys.version.startswith('2')
Found GPU at: /device:GPU:0 2.7.14 (default, Sep 23 2017, 22:06:14) [GCC 7.2.0]
There's a bunch of big blobs of code below - if you skip down to a cell below them and run the following, it will execute all cells before that one:
<Cmd>-<fn>-F8
(Note: you may not need to press fn key).
Click here to jump ahead to the right cell.
# basic image mgmt functions
!pip install scikit-image lucid
from __future__ import print_function
import numpy as np
import PIL
from IPython import display as ipyd
from lucid.misc.io import reading, showing
from skimage import io
# http://scikit-image.org/docs/dev/api/skimage.transform.html
from skimage import transform
from IPython import display as ipyd
# from https://stackoverflow.com/questions/39382412/crop-center-portion-of-a-numpy-image
def crop_center(img, cropx, cropy):
y, x, _ = img.shape
startx = x // 2 - (cropx // 2)
starty = y // 2 - (cropy // 2)
return img[starty:starty+cropy, startx:startx+cropx, :]
def square_shrink(img, dim):
smaller_dim = min(img.shape[:2])
cropped = crop_center(img, smaller_dim, smaller_dim)
small_cropped = transform.resize(cropped, (dim, dim, img.shape[2]), mode='constant')
return small_cropped
def load_img(fname, dim=None):
img = io.imread(fname)
if not dim:
return transform.resize(img, (img.shape[0], img.shape[1], img.shape[2]), mode='constant')
small_img = square_shrink(img, dim=dim)
return small_img
def upload_images():
from google.colab import files
uploaded = files.upload()
img_fnames = []
for fn in uploaded.keys():
print('User uploaded file "{name}" with length {length} bytes'.format(
name=fn, length=len(uploaded[fn])))
img_fnames.append(fn)
return img_fnames
Collecting scikit-image
Downloading https://files.pythonhosted.org/packages/26/5a/b95745a38ca32014f3604957a91f80d40a0aae9c6ea28a19048670be81b7/scikit_image-0.13.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (29.3MB)
100% |████████████████████████████████| 29.3MB 30kB/s eta 0:00:01
Requirement already satisfied: lucid in /Users/alook/code/lucid
Requirement already satisfied: scipy>=0.17.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/scipy-1.1.0rc1-py2.7-macosx-10.5-x86_64.egg (from scikit-image)
Collecting PyWavelets>=0.4.0 (from scikit-image)
Downloading https://files.pythonhosted.org/packages/da/2a/dd1b00eb1835e0f8d6f8b8f024fb77183cf1fe1d6910d95d48394cd80740/PyWavelets-0.5.2-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (4.8MB)
100% |████████████████████████████████| 4.8MB 175kB/s eta 0:00:01
Collecting networkx>=1.8 (from scikit-image)
Downloading https://files.pythonhosted.org/packages/11/42/f951cc6838a4dff6ce57211c4d7f8444809ccbe2134179950301e5c4c83c/networkx-2.1.zip (1.6MB)
100% |████████████████████████████████| 1.6MB 584kB/s eta 0:00:01
Requirement already satisfied: six>=1.7.3 in /opt/conda/miniconda2/lib/python2.7/site-packages (from scikit-image)
Collecting matplotlib>=1.3.1 (from scikit-image)
Downloading https://files.pythonhosted.org/packages/61/38/d70e8bf77d5cb27d5f3595edd0b3978825063feadd023786d2591e393e6e/matplotlib-2.2.2-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (13.7MB)
100% |████████████████████████████████| 13.7MB 79kB/s eta 0:00:01
Requirement already satisfied: pillow>=2.1.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/Pillow-5.1.0-py2.7-macosx-10.5-x86_64.egg (from scikit-image)
Requirement already satisfied: future in /opt/conda/miniconda2/lib/python2.7/site-packages (from lucid)
Requirement already satisfied: decorator in /opt/conda/miniconda2/lib/python2.7/site-packages (from lucid)
Requirement already satisfied: tensorflow in /opt/conda/miniconda2/lib/python2.7/site-packages/tensorflow-1.8.0rc0-py2.7-macosx-10.5-x86_64.egg (from lucid)
Requirement already satisfied: scikit-learn in /opt/conda/miniconda2/lib/python2.7/site-packages/scikit_learn-0.19.1-py2.7-macosx-10.5-x86_64.egg (from lucid)
Requirement already satisfied: numpy in /opt/conda/miniconda2/lib/python2.7/site-packages/numpy-1.14.2-py2.7-macosx-10.5-x86_64.egg (from lucid)
Requirement already satisfied: pyopengl in /opt/conda/miniconda2/lib/python2.7/site-packages/PyOpenGL-3.1.1a1-py2.7.egg (from lucid)
Requirement already satisfied: ipython in /opt/conda/miniconda2/lib/python2.7/site-packages (from lucid)
Collecting cycler>=0.10 (from matplotlib>=1.3.1->scikit-image)
Downloading https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
Requirement already satisfied: backports.functools-lru-cache in /opt/conda/miniconda2/lib/python2.7/site-packages (from matplotlib>=1.3.1->scikit-image)
Collecting subprocess32 (from matplotlib>=1.3.1->scikit-image)
Downloading https://files.pythonhosted.org/packages/b8/2f/49e53b0d0e94611a2dc624a1ad24d41b6d94d0f1b0a078443407ea2214c2/subprocess32-3.2.7.tar.gz (54kB)
100% |████████████████████████████████| 61kB 8.2MB/s eta 0:00:01
Collecting kiwisolver>=1.0.1 (from matplotlib>=1.3.1->scikit-image)
Downloading https://files.pythonhosted.org/packages/79/d8/94633718f3f77dcb638687a77ba199325a1cb158d2d4b00c9dc17f2b5c72/kiwisolver-1.0.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (110kB)
100% |████████████████████████████████| 112kB 7.9MB/s eta 0:00:01
Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/miniconda2/lib/python2.7/site-packages (from matplotlib>=1.3.1->scikit-image)
Requirement already satisfied: pytz in /opt/conda/miniconda2/lib/python2.7/site-packages (from matplotlib>=1.3.1->scikit-image)
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 (from matplotlib>=1.3.1->scikit-image)
Downloading https://files.pythonhosted.org/packages/6a/8a/718fd7d3458f9fab8e67186b00abdd345b639976bc7fb3ae722e1b026a50/pyparsing-2.2.0-py2.py3-none-any.whl (56kB)
100% |████████████████████████████████| 61kB 3.0MB/s eta 0:00:01
Requirement already satisfied: absl-py>=0.1.6 in /opt/conda/miniconda2/lib/python2.7/site-packages/absl_py-0.2.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: astor>=0.6.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/astor-0.6.2-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: backports.weakref>=1.0rc1 in /opt/conda/miniconda2/lib/python2.7/site-packages/backports.weakref-1.0.post1-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: enum34>=1.1.6 in /opt/conda/miniconda2/lib/python2.7/site-packages (from tensorflow->lucid)
Requirement already satisfied: gast>=0.2.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/gast-0.2.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: grpcio>=1.8.6 in /opt/conda/miniconda2/lib/python2.7/site-packages/grpcio-1.11.0-py2.7-macosx-10.5-x86_64.egg (from tensorflow->lucid)
Requirement already satisfied: mock>=2.0.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/mock-2.0.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: protobuf>=3.4.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/protobuf-3.5.2.post1-py2.7-macosx-10.5-x86_64.egg (from tensorflow->lucid)
Requirement already satisfied: tensorboard<1.8.0,>=1.7.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/tensorboard-1.7.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/termcolor-1.1.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: wheel in /opt/conda/miniconda2/lib/python2.7/site-packages (from tensorflow->lucid)
Requirement already satisfied: simplegeneric>0.8 in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: pickleshare in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: backports.shutil-get-terminal-size; python_version == "2.7" in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: pexpect; sys_platform != "win32" in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: pathlib2; python_version == "2.7" or python_version == "3.3" in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: traitlets>=4.2 in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: pygments in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: appnope; sys_platform == "darwin" in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: setuptools>=18.5 in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: futures>=2.2.0 in /opt/conda/miniconda2/lib/python2.7/site-packages (from grpcio>=1.8.6->tensorflow->lucid)
Requirement already satisfied: funcsigs>=1 in /opt/conda/miniconda2/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow->lucid)
Requirement already satisfied: pbr>=0.11 in /opt/conda/miniconda2/lib/python2.7/site-packages/pbr-4.0.2-py2.7.egg (from mock>=2.0.0->tensorflow->lucid)
Collecting bleach==1.5.0 (from tensorboard<1.8.0,>=1.7.0->tensorflow->lucid)
Downloading https://files.pythonhosted.org/packages/33/70/86c5fec937ea4964184d4d6c4f0b9551564f821e1c3575907639036d9b90/bleach-1.5.0-py2.py3-none-any.whl
Collecting html5lib==0.9999999 (from tensorboard<1.8.0,>=1.7.0->tensorflow->lucid)
Downloading https://files.pythonhosted.org/packages/ae/ae/bcb60402c60932b32dfaf19bb53870b29eda2cd17551ba5639219fb5ebf9/html5lib-0.9999999.tar.gz (889kB)
100% |████████████████████████████████| 890kB 928kB/s eta 0:00:01
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/miniconda2/lib/python2.7/site-packages/Markdown-2.6.11-py2.7.egg (from tensorboard<1.8.0,>=1.7.0->tensorflow->lucid)
Requirement already satisfied: werkzeug>=0.11.10 in /opt/conda/miniconda2/lib/python2.7/site-packages (from tensorboard<1.8.0,>=1.7.0->tensorflow->lucid)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/miniconda2/lib/python2.7/site-packages (from pexpect; sys_platform != "win32"->ipython->lucid)
Requirement already satisfied: scandir; python_version < "3.5" in /opt/conda/miniconda2/lib/python2.7/site-packages (from pathlib2; python_version == "2.7" or python_version == "3.3"->ipython->lucid)
Requirement already satisfied: ipython-genutils in /opt/conda/miniconda2/lib/python2.7/site-packages (from traitlets>=4.2->ipython->lucid)
Requirement already satisfied: wcwidth in /opt/conda/miniconda2/lib/python2.7/site-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->lucid)
Building wheels for collected packages: networkx, subprocess32, html5lib
Running setup.py bdist_wheel for networkx ... done
Stored in directory: /Users/alook/Library/Caches/pip/wheels/44/c0/34/6f98693a554301bdb405f8d65d95bbcd3e50180cbfdd98a94e
Running setup.py bdist_wheel for subprocess32 ... done
Stored in directory: /Users/alook/Library/Caches/pip/wheels/31/d3/4d/8151bf3fba7d27d0c9d03b78d4ff16097a8b62b298dc937c8d
Running setup.py bdist_wheel for html5lib ... done
Stored in directory: /Users/alook/Library/Caches/pip/wheels/50/ae/f9/d2b189788efcf61d1ee0e36045476735c838898eef1cad6e29
Successfully built networkx subprocess32 html5lib
Installing collected packages: PyWavelets, networkx, cycler, subprocess32, kiwisolver, pyparsing, matplotlib, scikit-image, html5lib, bleach
Found existing installation: html5lib 1.0.1
Uninstalling html5lib-1.0.1:
Successfully uninstalled html5lib-1.0.1
Found existing installation: bleach 2.1.3
Uninstalling bleach-2.1.3:
Successfully uninstalled bleach-2.1.3
Successfully installed PyWavelets-0.5.2 bleach-1.5.0 cycler-0.10.0 html5lib-0.9999999 kiwisolver-1.0.1 matplotlib-2.2.2 networkx-2.1 pyparsing-2.2.0 scikit-image-0.13.1 subprocess32-3.2.7
You are using pip version 9.0.2, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
--------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-1-9db9dc091a1c> in <module>() 5 6 from __future__ import print_function ----> 7 import numpy as np 8 import PIL 9 ImportError: No module named numpy
### Tensorflow deepdream model (inception5h)
# first, download a pretrained model to use
import os
if not os.path.exists('inception5h.zip'):
!wget https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip && unzip inception5h.zip
### Tensorflow Deepdream Code
# - Source: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/tutorials/deepdream
# - Original notebook: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/tutorials/deepdream/deepdream.ipynb
#
# Other references:
# - https://github.com/mtyka/deepdream_highres
#
# Source is inlcuded under Apache V2 License:
# - https://github.com/tensorflow/tensorflow/blob/master/LICENSE
# boilerplate code
from __future__ import print_function
import os
from io import BytesIO
import numpy as np
from functools import partial
import PIL.Image
from IPython.display import clear_output, Image, display, HTML
import tensorflow as tf
model_fn = 'tensorflow_inception_graph.pb'
# creating TensorFlow session and loading the model
graph = tf.Graph()
sess = tf.InteractiveSession(graph=graph)
with tf.gfile.FastGFile(model_fn, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
t_input = tf.placeholder(np.float32, name='input') # define the input tensor
imagenet_mean = 117.0
t_preprocessed = tf.expand_dims(t_input-imagenet_mean, 0)
tf.import_graph_def(graph_def, {'input':t_preprocessed})
layers = [op.name for op in graph.get_operations() if op.type=='Conv2D' and 'import/' in op.name]
feature_nums = [int(graph.get_tensor_by_name(name+':0').get_shape()[-1]) for name in layers]
print('Number of layers', len(layers))
print('Total number of feature channels:', sum(feature_nums))
# Helper functions for TF Graph visualization
def strip_consts(graph_def, max_const_size=32):
"""Strip large constant values from graph_def."""
strip_def = tf.GraphDef()
for n0 in graph_def.node:
n = strip_def.node.add()
n.MergeFrom(n0)
if n.op == 'Const':
tensor = n.attr['value'].tensor
size = len(tensor.tensor_content)
if size > max_const_size:
s = "<stripped %d bytes>"%size
tensor.tensor_content = str(s).encode('utf-8')
return strip_def
def rename_nodes(graph_def, rename_func):
res_def = tf.GraphDef()
for n0 in graph_def.node:
n = res_def.node.add()
n.MergeFrom(n0)
n.name = rename_func(n.name)
for i, s in enumerate(n.input):
n.input[i] = rename_func(s) if s[0]!='^' else '^'+rename_func(s[1:])
return res_def
def show_graph(graph_def, max_const_size=32):
"""Visualize TensorFlow graph."""
if hasattr(graph_def, 'as_graph_def'):
graph_def = graph_def.as_graph_def()
strip_def = strip_consts(graph_def, max_const_size=max_const_size)
code = """
<script>
function load() {{
document.getElementById("{id}").pbtxt = {data};
}}
</script>
<link rel="import" href="https://tensorboard.appspot.com/tf-graph-basic.build.html" onload=load()>
<div style="height:600px">
<tf-graph-basic id="{id}"></tf-graph-basic>
</div>
""".format(data=repr(str(strip_def)), id='graph'+str(np.random.rand()))
iframe = """
<iframe seamless style="width:800px;height:620px;border:0" srcdoc="{}"></iframe>
""".format(code.replace('"', '"'))
display(HTML(iframe))
# Visualizing the network graph. Be sure expand the "mixed" nodes to see their
# internal structure. We are going to visualize "Conv2D" nodes.
tmp_def = rename_nodes(graph_def, lambda s:"/".join(s.split('_',1)))
show_graph(tmp_def)
Number of layers 59 Total number of feature channels: 7548
## visualizing neuron activations
# start with a gray image with a little noise
img_noise = np.random.uniform(size=(224,224,3)) + 100.0
def showarray(a, fmt='jpeg'):
a = np.uint8(np.clip(a, 0, 1)*255)
f = BytesIO()
PIL.Image.fromarray(a).save(f, fmt)
display(Image(data=f.getvalue()))
def visstd(a, s=0.1):
'''Normalize the image range for visualization'''
return (a-a.mean())/max(a.std(), 1e-4)*s + 0.5
def T(layer):
'''Helper for getting layer output tensor'''
return graph.get_tensor_by_name("import/%s:0"%layer)
def render_naive(t_obj, img0=img_noise, iter_n=20, step=1.0):
t_score = tf.reduce_mean(t_obj) # defining the optimization objective
t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!
img = img0.copy()
for i in range(iter_n):
g, score = sess.run([t_grad, t_score], {t_input:img})
# normalizing the gradient, so the same step size should work
g /= g.std()+1e-8 # for different layers and networks
img += g*step
print(score, end = ' ')
clear_output()
showarray(visstd(img))
layer = 'mixed4d_3x3_bottleneck_pre_relu'
channel = 139 # picking some feature channel to visualize
render_naive(T(layer)[:,:,:,channel])
def tffunc(*argtypes):
'''Helper that transforms TF-graph generating function into a regular one.
See "resize" function below.
'''
placeholders = list(map(tf.placeholder, argtypes))
def wrap(f):
out = f(*placeholders)
def wrapper(*args, **kw):
return out.eval(dict(zip(placeholders, args)), session=kw.get('session'))
return wrapper
return wrap
# Helper function that uses TF to resize an image
def resize(img, size):
img = tf.expand_dims(img, 0)
return tf.image.resize_bilinear(img, size)[0,:,:,:]
resize = tffunc(np.float32, np.int32)(resize)
def calc_grad_tiled(img, t_grad, tile_size=512):
'''Compute the value of tensor t_grad over the image in a tiled way.
Random shifts are applied to the image to blur tile boundaries over
multiple iterations.'''
sz = tile_size
h, w = img.shape[:2]
sx, sy = np.random.randint(sz, size=2)
img_shift = np.roll(np.roll(img, sx, 1), sy, 0)
grad = np.zeros_like(img)
for y in range(0, max(h-sz//2, sz),sz):
for x in range(0, max(w-sz//2, sz),sz):
sub = img_shift[y:y+sz,x:x+sz]
g = sess.run(t_grad, {t_input:sub})
grad[y:y+sz,x:x+sz] = g
return np.roll(np.roll(grad, -sx, 1), -sy, 0)
def render_multiscale(t_obj, img0=img_noise, iter_n=10, step=1.0, octave_n=3, octave_scale=1.4):
t_score = tf.reduce_mean(t_obj) # defining the optimization objective
t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!
img = img0.copy()
for octave in range(octave_n):
if octave>0:
hw = np.float32(img.shape[:2])*octave_scale
img = resize(img, np.int32(hw))
for i in range(iter_n):
g = calc_grad_tiled(img, t_grad)
# normalizing the gradient, so the same step size should work
g /= g.std()+1e-8 # for different layers and networks
img += g*step
print('.', end = ' ')
clear_output()
showarray(visstd(img))
render_multiscale(T(layer)[:,:,:,channel])
k = np.float32([1,4,6,4,1])
k = np.outer(k, k)
k5x5 = k[:,:,None,None]/k.sum()*np.eye(3, dtype=np.float32)
def lap_split(img):
'''Split the image into lo and hi frequency components'''
with tf.name_scope('split'):
lo = tf.nn.conv2d(img, k5x5, [1,2,2,1], 'SAME')
lo2 = tf.nn.conv2d_transpose(lo, k5x5*4, tf.shape(img), [1,2,2,1])
hi = img-lo2
return lo, hi
def lap_split_n(img, n):
'''Build Laplacian pyramid with n splits'''
levels = []
for i in range(n):
img, hi = lap_split(img)
levels.append(hi)
levels.append(img)
return levels[::-1]
def lap_merge(levels):
'''Merge Laplacian pyramid'''
img = levels[0]
for hi in levels[1:]:
with tf.name_scope('merge'):
img = tf.nn.conv2d_transpose(img, k5x5*4, tf.shape(hi), [1,2,2,1]) + hi
return img
def normalize_std(img, eps=1e-10):
'''Normalize image by making its standard deviation = 1.0'''
with tf.name_scope('normalize'):
std = tf.sqrt(tf.reduce_mean(tf.square(img)))
return img/tf.maximum(std, eps)
def lap_normalize(img, scale_n=4):
'''Perform the Laplacian pyramid normalization.'''
img = tf.expand_dims(img,0)
tlevels = lap_split_n(img, scale_n)
tlevels = list(map(normalize_std, tlevels))
out = lap_merge(tlevels)
return out[0,:,:,:]
def render_lapnorm(t_obj, img0=img_noise, visfunc=visstd,
iter_n=10, step=1.0, octave_n=3, octave_scale=1.4, lap_n=4):
t_score = tf.reduce_mean(t_obj) # defining the optimization objective
t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!
# build the laplacian normalization graph
lap_norm_func = tffunc(np.float32)(partial(lap_normalize, scale_n=lap_n))
img = img0.copy()
for octave in range(octave_n):
if octave>0:
hw = np.float32(img.shape[:2])*octave_scale
img = resize(img, np.int32(hw))
for i in range(iter_n):
g = calc_grad_tiled(img, t_grad)
g = lap_norm_func(g)
img += g*step
print('.', end = ' ')
clear_output()
showarray(visfunc(img))
# Showing the lap_normalize graph with TensorBoard
lap_graph = tf.Graph()
with lap_graph.as_default():
lap_in = tf.placeholder(np.float32, name='lap_in')
lap_out = lap_normalize(lap_in)
show_graph(lap_graph)
render_lapnorm(T(layer)[:,:,:,channel])
render_lapnorm(T(layer)[:,:,:,65])
def render_deepdream(t_obj, img0=img_noise,
iter_n=10, step=1.5, octave_n=4, octave_scale=1.4):
t_score = tf.reduce_mean(t_obj) # defining the optimization objective
t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!
# split the image into a number of octaves
img = img0
octaves = []
for i in range(octave_n-1):
hw = img.shape[:2]
lo = resize(img, np.int32(np.float32(hw)/octave_scale))
hi = img-resize(lo, hw)
img = lo
octaves.append(hi)
# generate details octave by octave
for octave in range(octave_n):
if octave>0:
hi = octaves[-octave]
img = resize(img, hi.shape[:2])+hi
for i in range(iter_n):
g = calc_grad_tiled(img, t_grad)
img += g*(step / (np.abs(g).mean()+1e-7))
print('.',end = ' ')
clear_output()
showarray(img/255.0)
# #### TF RUN CODE
# # Picking some internal layer. Note that we use outputs before applying the ReLU nonlinearity
# # to have non-zero gradients for features with negative initial activations.
layer = 'mixed4d_3x3_bottleneck_pre_relu'
channel = 139 # picking some feature channel to visualize
render_naive(T(layer)[:,:,:,channel])
render_multiscale(T(layer)[:,:,:,channel])
# # Lower layers produce features of lower complexity.
# render_lapnorm(T('mixed3b_1x1_pre_relu')[:,:,:,101])
# # optimizing a linear combination of features often gives a "mixture" pattern.
# render_lapnorm(T(layer)[:,:,:,65]+T(layer)[:,:,:,139], octave_n=4)
# Semantic Dictionary setup from:
# TODO(look): rem`ove my workaround once the py3 bud gets patched
# - https://github.com/tensorflow/lucid/blob/master/notebooks/building-blocks/SemanticDictionary.ipynb
# Used under Apache V2 License:
# ---------------------------------------------------------------------
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# TODO(look): remove my "workaround" once the py3 bud gets patched
# !pip uninstall lucid
# !rm -rf /opt/lucid /content/src/lucid && git clone https://github.com/andrewlook/lucid.git /opt/lucid && cd /opt/lucid/ && git checkout py3
# ! cd /opt/lucid && python setup.py --quiet install
# !ls /opt/lucid/lucid/scratch/web/svelte.py
# !cp /opt/lucid/lucid/scratch/web/svelte.py .
# import svelte as lucid_svelte
!pip install --quiet lucid
import lucid.scratch.web.svelte as lucid_svelte
!npm install -g svelte-cli@2.2.0
import numpy as np
import tensorflow as tf
import lucid.modelzoo.vision_models as models
import lucid.optvis.render as render
from lucid.misc.io import show, load
from lucid.misc.io.showing import _image_url
/tools/node/bin/svelte -> /tools/node/lib/node_modules/svelte-cli/bin.js
/tools/node/lib
└── svelte-cli@2.2.0
%%html_define_svelte SemanticDict
<div class="figure">
<div class="input_image">
<div class="image" style="background-image: url({{image_url}}); z-index: -10;"></div>
<svg class="pointer_container" viewBox="0 0 {{N[0]}} {{N[1]}}">
{{#each xs as x}}
{{#each ys as y}}
<rect x={{x}} y={{y}} width=1 height=1
class={{(x == pos[0] && y == pos[1])? "selected" : "unselected"}}
on:mouseover="set({pos: [x,y]})"></rect>
{{/each}}
{{/each}}
</svg>
</div>
<div class="dict" >
{{#each present_acts as act, act_ind}}
<div class="entry">
<div class="sprite" style="background-image: url({{spritemap_url}}); width: {{sprite_size}}px; height: {{sprite_size}}px; background-position: -{{sprite_size*(act.n%sprite_n_wrap)}}px -{{sprite_size*Math.floor(act.n/sprite_n_wrap)}}px; --info: {{act.n}};"></div>
<div class="value" style="height: {{sprite_size*act.v/1000.0}}px;"></div>
</div>
{{/each}}
</div>
</div>
<style>
.figure {
padding: 10px;
width: 1024px;
}
.input_image {
display: inline-block;
width: 224px;
height: 224px;
}
.input_image .image, .input_image .pointer_constainer {
position: absolute;
width: 224px;
height: 224px;
border-radius: 8px;
}
.pointer_container rect {
opacity: 0;
}
.pointer_container .selected {
opacity: 1;
fill: none;
stroke: hsl(24, 100%, 50%);
stroke-width: 0.1px;
}
.dict {
height: 128px;
display: inline-block;
vertical-align: bottom;
padding-bottom: 64px;
margin-left: 64px;
}
.entry {
margin-top: 9px;
margin-right: 32px;
display: inline-block;
}
.value {
display: inline-block;
width: 32px;
border-radius: 8px;
background: #777;
}
.sprite {
display: inline-block;
border-radius: 8px;
}
.dict-text {
display: none;
font-size: 24px;
color: #AAA;
margin-bottom: 20px;
}
</style>
<script>
function range(n){
return Array(n).fill().map((_, i) => i);
}
export default {
data () {
return {
spritemap_url: "",
sprite_size: 64,
sprite_n_wrap: 1e8,
image_url: "",
activations: [[[{n: 0, v: 1}]]],
pos: [0,0]
};
},
computed: {
present_acts: (activations, pos) => activations[pos[1]][pos[0]],
N: activations => [activations.length, activations[0].length],
xs: (N) => range(N[0]),
ys: (N) => range(N[1])
},
helpers: {range}
};
</script>
Trying to build svelte component from html... svelte compile --format iife /tmp/svelte_KV5Sz3/SemanticDict_1bf1f72.html > /tmp/svelte_KV5Sz3/SemanticDict_1bf1f72.js svelte version 1.64.1 compiling ../tmp/svelte_KV5Sz3/SemanticDict_1bf1f72.html...
layer_spritemap_sizes = {
'mixed3a' : 16,
'mixed3b' : 21,
'mixed4a' : 22,
'mixed4b' : 22,
'mixed4c' : 22,
'mixed4d' : 22,
'mixed4e' : 28,
'mixed5a' : 28,
}
def googlenet_spritemap(layer):
assert layer in layer_spritemap_sizes
size = layer_spritemap_sizes[layer]
url = "https://storage.googleapis.com/lucid-static/building-blocks/googlenet_spritemaps/sprite_%s_channel_alpha.jpeg" % layer
return size, url
googlenet = models.InceptionV1()
googlenet.load_graphdef()
def googlenet_semantic_dict_from_url(layer, img_url):
img = load(img_url)
googlenet_semantic_dict(layer, img)
# ## **User facing constructor**
# Now we'll create a convenient API for creating semantic dictionary
# visualizations. It will compute the network activations for an image,
# grab an appropriate spritemap, and render the interface.
def googlenet_semantic_dict(layer, img):
# Compute the activations
with tf.Graph().as_default(), tf.Session():
t_input = tf.placeholder(tf.float32, [224, 224, 3])
T = render.import_model(googlenet, t_input, t_input)
acts = T(layer).eval({t_input: img})[0]
# Find the most interesting position for our initial view
max_mag = acts.max(-1)
max_x = np.argmax(max_mag.max(-1))
max_y = np.argmax(max_mag[max_x])
# Find appropriate spritemap
spritemap_n, spritemap_url = googlenet_spritemap(layer)
# Actually construct the semantic dictionary interface
# using our *custom component*
lucid_svelte.SemanticDict({
"spritemap_url": spritemap_url,
"sprite_size": 110,
"sprite_n_wrap": spritemap_n,
"image_url": _image_url(img),
"activations": [[[{"n": n, "v": float(act_vec[n])} for n in np.argsort(-act_vec)[:4]] for act_vec in act_slice] for act_slice in acts],
"pos" : [max_y, max_x]
})
googlenet_semantic_dict_from_url("mixed4d", "https://storage.googleapis.com/lucid-static/building-blocks/examples/dog_cat.png")
run the following, it will execute all cells before that one:
<Cmd>-<fn>-F8
(Note: you may not need to press fn key).
# Showing the lap_normalize graph with TensorBoard
lap_graph = tf.Graph()
with lap_graph.as_default():
lap_in = tf.placeholder(np.float32, name='lap_in')
lap_out = lap_normalize(lap_in)
show_graph(lap_graph)
# First run <Cmd>-<fn>-F8, to run all cells before this one.
# While you're waiting, you can hit <Shift>-Enter in this cell,
# and it will generate some output to verify that everything got set
# up correctly.
### Test deepdream just to make sure everything works
# Picking some internal layer. Note that we use outputs before applying the ReLU nonlinearity
# to have non-zero gradients for features with negative initial activations.
layer = 'mixed4d_3x3_bottleneck_pre_relu'
channel = 139 # picking some feature channel to visualize
render_naive(T(layer)[:,:,:,channel])
render_multiscale(T(layer)[:,:,:,channel])
render_lapnorm(T(layer)[:,:,:,channel])
# run this to display file upload form, and click "Choose Files"
# image_filenames = upload_images()
Saving j1.jpg to j1 (1).jpg Saving j2.jpg to j2 (1).jpg User uploaded file "j1.jpg" with length 64873 bytes User uploaded file "j2.jpg" with length 124776 bytes
# since images can be pretty large (and slow down the notebook to render them),
# it's handy to have some lower-res versions to quickly experiment with.
small_imgs = [load_img(f, dim=224) for f in image_filenames]
# use a helper from lucid to display the images neatly:
showing.images(small_imgs, labels=image_filenames)
# also load the full images, in case you want a high-res deepdream
large_imgs = [load_img(f) for f in image_filenames]
showing.images(large_imgs, labels=image_filenames, w=400)
# for the rest of the notebook to go smoothly, we'll keep a pointer to the first
# images that we loaded:
small_img_0 = small_imgs[0]
large_img_0 = large_imgs[0]
If you want to learn more about how to navigate Colab:
To get a brief overview of Colab's features:
# voiceover: back to slides <what is AI art?>
Definition #1: Making creative use of outputs from an AI Tool
Definition #2: Leveraging how AI represents information to deliberately craft an Effect
Definition #3: Exploring the Concepts within AI and what they mean for us
Moving forwards, we'll use Tool, Effect, and Concept to interpret different examples of art.
AI is a term used in so many contexts, that it can be confusing for an audience.
Most mentions of AI can be bucketed into one of two broad categories:
The Specialized kind (also known as Machine Learning).
We'll be focused on artistic applications of techniques that have sprung out of real-world research.
Many artistic tools come from people striving to understand algorithms affect our lives.
Machine Learning (usually) refers to a type of algorithm that:
Imagine we want to predict housing prices.
We'd probably start with a spreadsheet of prices we know to be true, alongside features for each house that we expect to be related to the price.

Source: medium.com/@kabab
# new slide
If we plot price vs. size on this toy dataset, we can start to see a slope

If we were to draw a line through these points, the "slope" would be what you multiply the size by in order to get the price (plus a "bias" term, if the slope doesn't meet the Y-axis right at zero).
The objective of an ML algorithm is to find a line to draw through these points that minimizes the error in its predictions.
# new slide
You can visualize the error as the distance from each real data point's Y coordinate to the prediction: the line's value at the corresponding X coordinate.

The way ML algorithms "learn" is defined by their objective.
In this case, they can start with a proposed line to draw, calculate the error, and make a next guess about what the best line could be.
This process gets repeated, and if everything goes according to plan, the errors decrease.

# consider basic neuron diagram...
Out of respect to artists everywhere, I'm not going to try to define it.
What's relevant here is that art can be defined by whoever is creating it.

"Fountain," Marcel Duchamp, 1917 Source: wikipedia
Definition #1: Making creative use of outputs from an AI Tool
Definition #2: Leveraging how AI represents information to deliberately craft an Effect
Definition #3: Exploring the Concepts within AI and what they means for us
These definitions actually mirror the progression of my own art practice.
I'll step through them in an attempt to share how my understanding of the space has evolved since I first started.
I started getting serious about learning ML sometime after starting to paint.
Around the same time, I took an art class about "Painting the Personal Narrative," and we needed to make a past / present series of still life paintings.
I started with a "present" painting, which I called "startup dinner" (a joking reference to my somewhat unhealthy eating habits left over from pre-Pinterest life).
# !cp drive/projects/startup_breakfast/3-breakfast-normal-orig.jpg drive/projects/whirlwind
# !cp drive/projects/startup_breakfast/3-breakfast-normal.jpg drive/projects/whirlwind
# !cp drive/projects/startup_breakfast/4-breakfast-dream.jpg drive/projects/whirlwind
breakfast_normal_full = 'drive/projects/whirlwind/3-breakfast-normal-orig.jpg'
breakfast_normal_cropped = 'drive/projects/whirlwind/3-breakfast-normal.jpg'
breakfast_dream_cropped = 'drive/projects/whirlwind/4-breakfast-dream.jpg'
small_img_0 = load_img(breakfast_normal_cropped, dim=224)
large_img_0 = load_img(breakfast_normal_cropped)
image_filenames = [
breakfast_normal_cropped,
breakfast_dream_cropped,
]
# since images can be pretty large (and slow down the notebook to render them),
# it's handy to have some lower-res versions to quickly experiment with.
small_imgs = [load_img(f, dim=224) for f in image_filenames]
# use a helper from lucid to display the images neatly:
showing.images(small_imgs, labels=image_filenames)
# also load the full images, in case you want a high-res deepdream
large_imgs = [load_img(f) for f in image_filenames]
#showing.images(large_imgs, labels=image_filenames, w=400)
# for the rest of the notebook to go smoothly, we'll keep a pointer to the first
# images that we loaded:
small_img_0 = small_imgs[0]
large_img_0 = large_imgs[0]
When we were asked to make a painting of the "past," the prompt was to make it "nightmarish."
I had another photo chosen already, which i jokingly titled "startup breakfast".
The only problem was, I didn't know how to make a dream effect.
ipyd.Image(filename=breakfast_normal_full, width=600)
Thankfully, code for making deepdreams had just been announced by Google Research.

small_new_img = render_deepdream(tf.square(T('mixed4c')), small_img_0*255)
new_img = render_deepdream(tf.square(T('mixed4c')), large_img_0*255)



# TODO convo explaining this to someone
# TODO neural nets
# TODO weight updates / backprop basics
# TODO explaining dream / featureviz
Even without much context on how it works, Pix2Pix is fun to play with and yields wierd results.
We'll look at some more in-depth approaches later (time permitting), but for now this is just fun.

Based on: "Image-to-Image Translation with Conditional Adversarial Networks", Isola et al, 2017
# Visualizing the network graph. Be sure expand the "mixed" nodes to see their
# internal structure. We are going to visualize "Conv2D" nodes.
tmp_def = rename_nodes(graph_def, lambda s:"/".join(s.split('_',1)))
show_graph(tmp_def)
mixed4d = 'mixed4d_3x3_bottleneck_pre_relu'
mixed4d_65 = render_lapnorm(T(mixed4d)[:,:,:,65])
# Lower layers produce features of lower complexity.
mixed3b_101 = render_lapnorm(T('mixed3b_1x1_pre_relu')[:,:,:,101])
# optimizing a linear combination of features often gives a "mixture" pattern.
combo__mixed4d_65__mixed3b_101 = render_lapnorm(T(mixed4d)[:,:,:,65]+T(mixed4d)[:,:,:,139], octave_n=4)
#### TF RUN CODE
render_lapnorm(T(layer)[:,:,:,65])
# Lower layers produce features of lower complexity.
render_lapnorm(T('mixed3b_1x1_pre_relu')[:,:,:,101])
# optimizing a linear combination of features often gives a "mixture" pattern.
render_lapnorm(T(layer)[:,:,:,65]+T(layer)[:,:,:,139], octave_n=4)
For example, recent generative algorithms have rapidly improved their ability both to learn latent spaces, and to generate images from any point in these latent spaces.
In fact, those spooky faces earlier came from a visualization of Eigenfaces, which sought to "learn" how to represent faces in a "latent space".

Researchers have built tools to allow visual navigation of latent spaces in order to gain fine-grained control over the generated artifacts.

When we interpolate between points in the space, generating images at each point along the way, it's striking that (almost) every point in between looks like a person.
One can't but help thinking that these algorithms are beginning to learn some fundamental truths about what we as humans all have in common.

Source: youtube
Paper: Progressive Growing of GANs for Improved Quality, Stability, and Variation, Karras et al, 2018
Among other things, this could encompass exploring AI's relationships to:
# TODO sketchrnn / DRAW / paint?